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Abstract 

Background: In middle income countries the number of trained health technology assessment specialists is limited 
and the public budget for health technology assessment is considerably lower compared to developed countries. 
These countries therefore must develop their own solutions to improve the quality and efficiency of health 
technology assessment implementation in reimbursement decisions. Our study aimed to develop a scientifically 
rigorous and detailed appraisal checklist for economic evaluations of pharmaceuticals in the single health 
technology assessment process. 

Methods: The research design entailed a review of economic evaluations, submitted for reimbursement of 
pharmaceuticals, by two independent academic reviewers to identify the most common methodological problems. 
Fifty economic evaluations submitted in 2007-2008, randomly selected by the Health Technology Assessment Office 
served as data sources. The new checklist was developed by an iterative working process: first by assessing ten 
economic evaluations, then improving the checklist by generating new question items, then employing the 
improved checklist to assess the next ten economic evaluations. After appraising 25 documents, the reviewers 
reconciled their opinions and improved the checklist with the researchers of the Health Technology Assessment 
Office during an expert panel discussion. The reviewers scrutinized the second 25 economic evaluations, after 
which the expert panel finalized the checklist with consensus. 

Results: The final checklist consists of 91 yes or no questions in 1 1 main topics concerning comparator selection, 
efficacy, effectiveness, costs, sensitivity analysis, methodological approach, transparency, and interpretation of 
results. The new checklist is based on current Hungarian evaluation practice. As the published checklist will be part 
of the official single health technology assessment process of pharmaceuticals, submitters will be able to assure the 
quality of their economic evaluation. 

Conclusions: The transparent critical appraisal method should improve the consistency of pharmaceutical 
reimbursement decisions and facilitate the utilization of economic evaluations in other fields of health care 
decision-making in other Central-Eastern European countries. 
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Background 

Compared to high-income European countries, middle 
income Central and Eastern European (CEE) countries 
have worse health status [1,2]. To improve the health of 
population, these countries have even more limited 
resources than Western European countries. As the 
value based price of innovative health technologies is 
adjusted to top 5 European countries, the price is often 
relatively too high in countries with lower income [3], 
Consequently the need for justification of cost- 
effectiveness for expensive technologies is even greater 
in CEE countries. Unfortunately the number of trained 
health economists and health technology assessment 
(HTA) specialists is limited and the public budget for 
HTA is lower than in developed countries [4-6]. There- 
fore middle income countries must develop their own 
solutions to improve the quality and efficiency of HTA 
implementation in reimbursement decisions. 

The first necessity is a development of a guideline for 
economic evaluations. It is, however, difficult to develop 
guideline which is perfectly relevant for all different kind 
of health technologies and services. For example, the 
HTA approach for medical devices is different from 
pharmaceuticals due to the limited availability of rando- 
mised controlled trials (RCTs), and other methodological 
problems such as multiple indications, frequent product 
modifications, learning curves, and high fixed cost [7]. 
HTA guidelines are usually broad enough to cover all 
different aspects [8,9], however the National Institute for 
Health and Clinical Excellence also developed focused 
guidelines for diagnostics [10] and devices [11]. 

The second necessity of quality assurance in HTA is 
the use of critical appraisal checklists especially in those 
countries where single HTA process is implemented. In 
the current examples of this process this means that the 
manufacturer of the technology prepares the assessment 
of the technology. The public HTA office cannot prepare 
an independent assessment (double technology assess- 
ment) and is only responsible for the critical appraisal of 
the submitted material. In countries with single HTA 
process, the most critical question is how to ensure the 
appropriateness, the homogeneity and the transparency 
of the critical appraisal. There are two different 
approaches: The first focuses on the transparent descrip- 
tion of the appraisal methodology, the second focuses 
more on disclosing details of each individual case. As 
submissions may contain confidential information (i.e. 
proposed price of the technology), the first approach can 
be easily implemented in any countries. 

There are many available international checklists 
[12-15], however, they are often not detailed enough to 
address specificities of long HTA submissions or not 
adjusted to the country specific methodological pro- 
blems. Therefore authors suggest that for critical 



appraisal of single technology assessment, detailed coun- 
try and technology specific checklists should be devel- 
oped based on the review of existing local HTA practices. 

In Hungary cost-effectiveness and budget impact ana- 
lyses have been mandated in the reimbursement process 
of new pharmaceuticals since 2003. Guidelines for eco- 
nomic evaluations were published [16] in 2002, but have 
never been updated. These guidelines covered all health 
care interventions therefore they were not specific for 
pharmaceuticals and not targeted to reimbursement 
questions. The first six years experience of mandatory 
fourth hurdle (i.e. local cost-effectiveness evidence) for 
new pharmaceuticals indicated that the quality of eco- 
nomic evaluations submitted in pharmaceutical reim- 
bursement dossiers and the quality of critical appraisal 
by the HTA Office varied considerably. Similar trends 
have been observed in Canada [17], Latin America [18], 
and Sweden [19]. 

Therefore in 2009, an expert panel was established to 
develop a detailed, publicly available, scientifically rigor- 
ous, and policy-relevant Hungarian critical appraisal 
checklist to improve the quality of economic evaluations 
and budget impact analyses submitted for single health 
technology assessment in pharmaceutical reimburse- 
ment applications and to reduce the heterogeneity of 
appraisals. This paper describes the methodological ap- 
proach of checklist development, and presents the final 
checklist. 

Methods 

The expert panel consisted of two independent academic 
experts, who scrutinized previous submissions and 
developed a new draft checklist, and members of the 
HTA Office who supervised the project and approved 
new items in the checklist. To maintain the full support 
of decision-makers in the Ministry of Health and the 
National Health Insurance Fund the project had no 
intention to revise or comment on previous reimburse- 
ment decisions. Overall, 50 consecutive economic eva- 
luations that had been submitted for reimbursement of 
pharmaceuticals in 2007-2008 were selected by the HTA 
Office to be scrutinized by the two independent aca- 
demic reviewers for the most common methodological 
problems. The first ten submissions were assessed using 
the previously translated Critical Appraisal Skills 
Programme (CASP) questionnaire [20] in the form of a 
spreadsheet independently completed by each reviewer. 
In the process, several new items were added to reflect 
common methodological or technical problems. The 
spreadsheet included a column listing the questions 
from the new checklist (one per row), and each eco- 
nomic evaluation had a separate column to record the 
reviewers' answers and comments. The assessments of 
the first ten submissions were then reconciled by the 
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reviewers. The format of the CASP checklist was also 
changed; as many of the new question items were not 
relevant for all submitted economic evaluations, a "not 
relevant" option was added to "yes" or "no" option. For 
the next ten submissions, a draft version of the new 
extended checklist was employed (Figure 1). Each new 
technical problem observed generated a new question 
item under the assumption that these questions could 
prevent future occurrences of these problems. Conse- 
quently, the development process was iterative: as an- 
other generally applicable problem was identified the 
reviewers added a new question item to the draft ques- 
tionnaire. After 25 economic evaluations were assessed, 
the opinions of the reviewers were again reconciled and 
the checklist was revised through an expert panel dis- 
cussion with the entire staff of the HTA Office. Subse- 
quently, the reviewers scrutinized the second 25 
economic evaluations, again extending the checklist 
with further question items. The final checklist was 
achieved through a consensus decision of the expert 
panel. 

The Hungarian Critical Appraisal Checklist is not a 
scorecard; it only lists the most relevant questions for fa- 
cilitating the appraisal process. While quantitative ques- 
tionnaires (i.e., those with scoring) can be validated by 
statistical methods, it is less obvious how to validate a 
qualitative critical appraisal checklist. The development 
process described here incorporated three steps to im- 
prove validity. First, every new question item was incor- 
porated only through consensus of the expert panel. 
Second, in a 14-month probation period, between 
September 2009 and November 2010, the checklist was 
used by the public HTA office in the appraisal of 80 new 
pharmaceutical reimbursement submissions. Researchers 
of the HTA Office confirmed the appropriateness of the 
checklist and reported no need for further amendments. 
Finally, an independent opponent was given the oppor- 
tunity to publicly challenge the pre-circulated checklist 



during a two-hour meeting of the Hungarian Health 
Economics Association dedicated to the public discus- 
sion of the checklist. This followed an approach similar 
to that of the Health Economist's Study Group in the 
United Kingdom [21]. 

Results 

The final Hungarian Critical Appraisal Checklist 
(Additional file 1) has two major themes: the first 
addresses economic evaluations, whereas the second 
concentrates on budget impact analyses. Economic eva- 
luations are the subject of 80 yes or no questions in ten 
major sections; 11 questions relate specifically to budget 
impact analysis (see Table 1). 

The ten sections addressing economic evaluations 
cover core ideas related to the selection of comparators, 
effectiveness, costs, sensitivity analysis, methodological 
approach, transparency, and interpretation of results. To 
reflect the possible variations in study design of eco- 
nomic evaluations, separate alternative sections were 
developed for cost-minimization analyses and modeling 
approaches of cost-effectiveness analyses (including deci- 
sion tree, Markov and simulation models). This separ- 
ation allows the incorporation of methodology-specific 
issues into the checklist. Interestingly, none of the 50 re- 
imbursement dossiers reviewed included economic ana- 
lyses conducted alongside prospective clinical trials; this 
was mainly due to the fact that Hungary is too small 
market for conducting adequately powered prospective 
trials for local reimbursement purposes. 

When the checklist is used for critical appraisal, 
reviewers may exclude non-relevant question items. If 
the problem raised by a question is relevant and has 
been addressed appropriately in the economic analysis of 
pharmaceuticals, the question should be answered "yes". 
If a "yes" answer cannot be justified in the case of a rele- 
vant question, the response should be "no". The authors 
therefore encourage the inclusion of explanatory 
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Table 1 Structure of the checklist 

Topic # of items 

1. Economic Evaluation 80 

1.1. Filter Questions 2 

1.2. Research Question (relevance, comparator, 2 
financing protocol) 

1.3. Health Benefit 25 

1.3.1. Source of Scientific Evidence 7 

1.3.2. Evaluation of Relative Effectiveness in Case 10 
of Indirect Comparison 

1.3.3. Magnitude of Health Benefit 8 

1 .4. Cost 8 

1.5. Time Horizon, Discounting 3 

1 .6 Alternative Sections for Methodology 22 

1.6.1. Cost-Minimization Analysis 3 

1.6.2. Cost Effectiveness Analyses 19 

1.6.2.1. Decision Tree Model 4 

1.6.2.2. Markov Model 7 

1.6.2.3. Simulation Model 8 

1 .7. Decision Rule 3 

1.8. Sensitivity Analysis 6 

1.9. General Methodology: Adequacy and Transparency 5 

1.10. Interpretation regarding the Economic Evaluation 4 

2. Budget Impact Analysis 1 1 
Total checklist 91 



comments for all answers in the "explanation" column. 
Sub-questions in brackets that follow some questions 
also prompt explanations. 

An important issue when developing checklists is 
the distinction between the positive aspect of reporting 
(e.g. did they apply discounting?) and normative aspects 
of methods (e.g. did they discount by a rate of 5% per 
annum in real terms?). We took an intermediate ap- 
proach, in several cases we considered the normative 
aspects without clarifying what is the only appropriate 
approach (e.g. did they apply the appropriate discount 
rate?), as the correct approach can be different for par- 
ticular technologies. 

In use, a higher proportion of "yes" answers would in- 
dicate the appropriateness of the economic evaluation 
and budget impact analysis for inclusion in the reim- 
bursement dossier. However, the questions are not equal 
and not weighted; therefore, the critical appraisal check- 
list cannot transform the assessment of the quality of 
the single technology appraisal into an automatic 
process. There are twofold benefits from the checklist. 
At first to guide those who prepare submissions on what 
are the most critical methodological and technical ques- 
tions of the HTA report, secondly to make sure that 
these important factors are not missed in the critical 



appraisal process, therefore the heterogeneity of apprai- 
sals is reduced. 

Discussion 

Certain countries assure the quality of HTA in the reim- 
bursement decision process by improving the details of 
their guidelines; however in these cases the guideline 
may not be relevant for all different technologies 
(e.g. pharmaceuticals, medical devices, screening proce- 
dures etc.). For middle income CEE countries with lim- 
ited budget and capacity, general guidelines and detailed 
supplementary checklists for each kind of health tech- 
nologies could be an optimal solution. As a first step, we 
presented a prototype how to prepare such a scientific- 
ally rigorous appraisal tool specifically developed for 
pharmaceutical technologies based on the quality assess- 
ment of economic evaluations in previous reimburse- 
ment dossiers. The existence of such tool, however, 
could not substitute the overdue updating of the 
Hungarian guidelines for economic evaluation or the de- 
velopment of a Reference Guide to those making sub- 
missions. Development of similar checklists for medical 
devices or other medical technologies should also be 
considered. Both the methodology of development and/ 
or the actual questions of the checklist might be of inter- 
est for other CEE countries with single HTA to assure 
the better use of economic evaluations in the reimburse- 
ment of pharmaceuticals. For CEE countries where HTA 
was recently introduced, this checklist could serve as a 
draft to develop their own country specific question- 
naire, with the adaptation of the iterative methodology 
described in this paper. 

This Hungarian checklist was developed to detect and 
consequently to prevent potential methodological and 
technical problems seen in economic evaluations in 
pharmaceutical reimbursement submissions. Identifica- 
tion of these issues requires advanced skills; thus, 
authors want to emphasize how important the qualifica- 
tions of the assessors are [22]. However, based on the 14 
month probation period, such a detailed checklist is a 
useful aid even for new and relatively inexperienced 
associates of the HTA Office. 

There is a trade-off between the sensitivity and simpli- 
city of critical appraisal checklists (Table 2 compares 
some well known international checklists with the 
Hungarian Critical Appraisal Checklist). Shorter ques- 
tionnaires are suitable for a rapid assessment of eco- 
nomic evaluations, e.g., a review of scientific papers [23]. 
However, neither the shorter [24] nor the longer [15] 
international critical appraisal checklists are detailed 
enough to reliably filter all problems observed in health 
economic analyses in reimbursement submissions of 
pharmaceuticals, particularly as inappropriate decisions 
by the authorities may result in serious health and 
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Table 2 Comparison of the Hungarian Critical Appraisal Checklist to some other international checklists 



Name of the checklist 


Drummond/CASP 
checklist 


British Medical 
Journal (BMJ) 
checklist 


Consensus Health 
Economic Criteria 
(CHEC) list 


Quality of Health 
Economic Studies 
(QHES) grading system 


Hungarian Critical 
Appraisal Checklist 


Number of questions 


10/ 12 


35 


19 


16 


91 


Answer options 


YES/NO/Can't tell 


YES/NO/Not appropriate 


YES/NO 


YES/NO 


YES/NO/Not relevant 


Quantitative assessment 
(including weighting 
different items) available? 


no 


no 


no 


yes 


no 


Reference 


[20,24] 


[1 5,22] 


[14,22] 


[13,22] 


NA. 



financial consequences. Furthermore, the more general 
questions seen on the shorter lists can be interpreted dif- 
ferently, further eroding their usefulness for assessors. The 
majority of existing questionnaires are based on the 
current sophisticated skill set and practice of developed 
countries, therefore may not be suitable in middle income 
countries due to limited capacity of experts and hetero- 
geneity of submissions. The time required to answer a 
detailed questionnaire (such as the Hungarian Critical Ap- 
praisal Checklist with 91 items) must be acknowledged, 
but the thorough revision of data and methodology in a 
public HTA appraisal requires several days for each sub- 
mission anyway. In addition, compared with scientific 
publications, economic evaluations in reimbursement sub- 
missions are lengthier, and include more details about 
assumptions, input data, and study methodology [25]. 

Some unique items of the Hungarian Critical Ap- 
praisal Checklist should be highlighted. Specific ques- 
tions are dedicated to therapeutic guidelines, the 
financing protocol (i.e., whether the therapy is reim- 
bursed for first-line use or only second- or third-line), 
and the level of reimbursement (due to the complexity 
of the Hungarian public financing system). Such ques- 
tions were not common in previous checklists. Based on 
50 submissions, a question is dedicated to prevent in- 
appropriate selection of studies favouring the investiga- 
tional technology. Assessment of relative effectiveness in 
indirect comparison of pharmaceuticals has gained in- 
creasing importance recently [26], as the comparator in 
pivotal clinical trials may not be policy relevant in all 
countries, e.g., the comparator may not be reimbursed 
or widely used. As a consequence, ten specific questions 
have been included to evaluate the methodology of rela- 
tive effectiveness in case an indirect comparison is 
employed in submissions. A recurring methodological 
error was the calculation of daily therapeutic drug cost 
from the perspective of the third-party payer. Unless the 
full public price is employed in the calculation, the most 
cost-effective scenario from the payer's perspective 
would be a 0% reimbursement, i.e., no increase in drug 
costs with additional health gain. By requesting the 
graphical structure and detailed description of decision 
tree and Markov models and the transparent description 



of all input data, the HTA Office should be able to re- 
construct some of the economic models [27]. This prac- 
tice prevents the assessment of cost-effectiveness based 
on black box economic models, which was common ap- 
proach previously. In some extreme submissions, a long 
general introduction was accompanied by a brief eco- 
nomic evaluation; therefore, a question was dedicated to 
address the proportionality of economic evaluation. Ex- 
plicit declaration of known limitations in the submitted 
economic evaluation is also not a regular part of critical 
appraisal checklists; the Hungarian checklist addresses 
this issue. According to Ramsberg et al. [19], checklists 
tend to be too general to pick up fine distinctions of spe- 
cific models. 

A limitation of the Hungarian Critical Appraisal 
Checklist could be that its development is based on the 
reappraisal of only 50 submissions, and thus may not re- 
flect all potential problems. The Hungarian checklist 
does not employ a scoring system; consequently it can- 
not rank different economic evaluations quantitatively. 
Unless items are explicitly weighted, the implicit general 
assumption is that all questions are treated as equally 
important [28]. Compared with previous checklists with 
a scoring system, the Hungarian Critical Appraisal 
Checklist, as a qualitative tool, could not be validated 
with statistical methods. Certain steps of the methodo- 
logical approach employed in this research were similar 
to methods in previous publications. Gerkens et al. [22] 
appraised nine economic evaluations to compare three 
different checklists (British Medical Journal (BMJ) 
checklist, Consensus Health Economic Criteria (CHEC) 
list, Quality of Health Economic Studies (QHES) grading 
system). The development of a scoring system by 
Gonzalez-Perez [23] was based on a 50-study sample. 
Ramsberg et al. [19] scrutinized a sample of 20 submis- 
sions in Sweden (2002-2003). Similarly to our approach, 
these studies applied at least two independent reviewers. 

A further development of the Hungarian checklist 
could be to assign weights to each question. However, 
the aim of this checklist is not to score submissions by 
counting the number of "yes" answers, but to avoid 
methodological and technical errors in future submis- 
sions by using this instrument when compiling 
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pharmaceutical reimbursement dossiers. Although the 
checklist is based on current Hungarian practice, the 
authors could dedicate questions only to those problems 
that occurred in the 50 assessed HTA documents. The 
authors recommend that the checklist be revised period- 
ically, approximately every two years. Continuous devel- 
opment of the checklist can prevent to be grounded in 
the current methodology. E.g. the current methodo- 
logical guidelines for economic evaluations in Hungary 
does not necessitate the use of probabilistic sensitivity 
analysis (PSA), once it becomes recommended, the 
checklist can be extended with new items on PSAs. As 
the Hungarian methodological guidelines determine nor- 
mative aspects of economic analyses and some of these 
items (e.g. discount rate, cost effectiveness threshold) are 
subject to change in the forthcoming guidelines, the crit- 
ical appraisal checklist does not mention any normative 
aspects to ensure full harmonisation with the guidelines. 

Conclusions 

The new Hungarian Critical Appraisal Checklist is 
detailed enough to address the most common problems 
in local economic evaluations and budget impact ana- 
lyses submitted in reimbursement dossiers by pharma- 
ceutical companies. The published checklist will be used 
officially by the HTA Office in the pharmaceutical's sin- 
gle health technology assessment process. As it will be 
in the public domain, application of the checklist should 
improve the consistency of the appraisal process and 
consequently encourage pharmaceutical companies to 
assure the quality of their submitted economic evalua- 
tions. Overall, the use of a transparent method of single 
technology assessment should improve the appropriate- 
ness of pharmaceutical reimbursement decisions. 

The authors believe that the Hungarian Critical Ap- 
praisal Checklist will prove to be a significant step towards 
the better use of economic evaluations in the reimburse- 
ment of pharmaceuticals and by adopting the develop- 
ment methodology or certain questions, may have policy 
implications for several other Central-Eastern European 
middle income countries. It is noteworthy, especially in 
Central-Eastern Europe, that a public authority has been 
willing to develop and publish scientifically rigorous cri- 
teria for decision-making purposes. We are not aware of 
any published critical appraisal checklists specifically 
developed for pharmaceuticals based on the HTA practice 
and skill set of middle income countries. 

Additional file 



Additional file 1: Hungarian critical appraisal checklist. 
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